Device and a method for merging candidate areas

ABSTRACT

A device and method merge a first candidate area relating to a candidate feature in a first image and a second candidate area relating to a candidate feature in a second image. The first and second images have an overlapping region, and at least a portion of the first and second candidate areas are located in the overlapping region. An image overlap size is determined indicating a size of the overlapping region of the first and second images, and a candidate area overlap ratio is determined indicating a ratio of overlap between the first and second candidate areas. A merging threshold is then determined based on the image overlap size, and, on condition that the candidate area overlap ratio is larger than the merging threshold, the first candidate area and the second candidate area are merged, thereby forming a merged candidate area.

FIELD OF INVENTION

The present invention relates to merging candidate areas, andspecifically to merging a candidate area relating to a feature in afirst image and a second candidate area relating to a feature in asecond image.

TECHNICAL BACKGROUND

In some image processing applications candidate areas in the image areidentified, wherein the candidate areas relate to features, such asobjects, segments of an object or other, in the image. For example, suchcandidate areas may be candidate bounding boxes relating to objectdetections in the image, wherein the candidate bounding boxes may havebeen derived by means of an object detection algorithm, e.g., usingneural networks. In other examples, such candidate areas may becandidate polygons or pixel masks derived by means of an instancesegmentation algorithm, or pixel masks derived from a motion detectionalgorithm. Once candidate areas in the image have been identified,merging algorithms may be performed in order to determine whether or nottwo candidate areas should be merged into a single candidate area. Suchdetermining is typically based on threshold for a ratio of overlapbetween the two candidate areas, i.e., how much do the candidate areasoverlap in relation to their total sizes. For a ratio of overlap overthe threshold, the candidate areas are merged, and for a ratio ofoverlap below the threshold, the candidate areas are not merged. The aimof such merging algorithms is that determining to merge two candidateareas should correspond to cases where the two candidate areas in factrelate to the same feature in the image. Such merging algorithms havebeen developed for scenarios when the candidate areas have beenidentified in a single continuous image. However, there are scenarioswhen such algorithms are not successful in identifying all relevantsituations when the candidate areas should be merged, i.e., when thecandidate areas in fact relate to the same feature in the image. Oneexample of such a scenario is when the candidate areas have beenidentified in a discontinuous image captured by a single image sensor,wherein two portions (handled as two separate images when identifyingcandidate areas) of the discontinuous image have an overlapping region,which in this case is a region captured by a same region of the singleimage sensor. Another example of such a scenario is when the candidateareas have been identified in two separate images captured by twoseparate image sensors, wherein the two images include an overlappingregion, which in this case is a region representing a portion of a scenecaptured by both of the two separate image sensors.

SUMMARY

Facilitating enhanced identification and merging of two candidate areasin relation to two respective images, where in the two respective imageshave an overlapping region, would be beneficial.

According to a first aspect, a method for merging a first candidate arearelating to a candidate feature in a first image and a second candidatearea relating to a candidate feature in a second image is provided. Thefirst image and the second image have an overlapping region, and atleast a portion of the first candidate area and at least a portion ofthe second candidate area are located in the overlapping region. Themethod comprises determining an image overlap size indicating a size ofthe overlapping region of the first image and the second image;determining a candidate area overlap ratio indicating a ratio of overlapbetween the first candidate area and the second candidate area;determining a merging threshold based on the image overlap size; and oncondition that the candidate area overlap ratio is larger than themerging threshold, merging the first candidate area and the secondcandidate area, thereby forming a merged candidate area.

By “candidate feature” is meant an object, part of an object, segment ofan object, a portion where there is a movement or any other feature ofan image for which a candidate area has been identified in precedingprocessing and being input to the method according to the first aspect.

By “image” in relation to a first image and a second image is meant thatthe first image and the second image have been handled separately whenidentifying candidate areas. The first image and the second image mayhave been captured by a respective one of a first image sensor and asecond image sensor. In alternative, the first image and the secondimage may be a first portion and a second portion, respectively, of animage captured using a single image sensor. In the latter case, thefirst image and the second image may be transformations of the firstportion and the second portion, respectively.

By “overlapping region” in relation to the first and second imageshaving an overlapping region is meant that there is a region in oneimage of the first and second images that is included also in the otherimage with or without transformation, or that there is a region in oneimage of the first and second images that is a representation of a sameportion of a scene as a region in the other image. In the former case,the first and second images may for example be transformations ofportions of a wide-angle view of a scene in order to reduce distortionsof feature size relations. In the latter case, the first and secondimages may be images captured by first and second image sensors,respectively, wherein the first and second image sensors capturerespective but partially overlapping portions of the scene.

By “ratio” in relation to the ratio of overlap between the firstcandidate area and the second candidate area is meant any kind ofnormalization in relation to sizes of the first and second candidateareas.

For a first candidate area identified in a first image and a secondcandidate area identified in a second image, wherein the first image andthe second image have an overlapping region, the overlap between thefirst candidate area and the second candidate area is limited in size bythe size of the overlapping region. For a candidate feature of which aportion is in the first image and a portion is in the second image therewill be a common portion of the candidate feature that will be in theoverlapping region, i.e., that common portion will appear in both thefirst image and in the second image. As a first candidate area relatingto the candidate feature identified in the first image and a secondcandidate area relating to the candidate feature identified in thesecond image can only overlap in the overlapping region. To accommodatefor this limitation, the merge threshold is according to the firstaspect is determined based on the size of the overlapping region.

By determining the merging threshold based on the image overlap size therisk of not merging the first candidate area and the second candidatearea even when they relate to the same candidate feature is reduced.Hence, the risk is reduced that the same candidate feature in the twoseparate images risk be identified as two separate candidate features.

The merging threshold may be determined to be increasing with the imageoverlap size. For example, the merging threshold may be determined to beproportional to the image overlap size. Hence, for a given overlap size,the merging threshold will be determined to be the given overlap sizemultiplied with a proportionality constant.

The candidate area overlap ratio may be determined as an intersection ofthe first candidate area and the second candidate area divided by theunion of the first candidate area and the second candidate area. Thismeasure of the candidate area overlap ration is commonly referred to asIntersection over Union (IoU).

The first candidate area and the second candidate area may be identifiedin coordinates of a combined image comprising the first image and thesecond image. This may for example relate to a case where the firstimage and the second image have been captured by a first image sensorand a second image sensor, respectively, wherein the first image sensorand the second image sensors capture respective but partiallyoverlapping portions of a scene.

The first image may be a first transformed view of a first portion of afisheye image and the second image may be a second transformed view of asecond portion of the fisheye image, and the first candidate area andthe second candidate area may be identified in coordinates of thefisheye image or in coordinates of some other coordinate system. Thisrelates to a case where the first image and the second image are a firstportion and a second portion, respectively, of an image captured using asingle image sensor and wherein, the first image and the second imageare transformations of the first portion and the second portion,respectively. Hence, the first candidate area and the second candidatearea having been identified in the first image and the second image,respectively, have been transformed back such that they are identifiedin the coordinates of the fisheye image or in coordinates of some othercoordinate system.

The merged candidate area may consist of the union of the firstcandidate area and the second candidate area. Alternatives areenvisaged. If, for example, the merged candidate area should have aparticular shape, the merged candidate area may be an area of theparticular shape comprising the first candidate area and the secondcandidate area. For example, the merged candidate area may be thesmallest area of the particular shape comprising the first candidatearea and the second candidate area.

The first and second candidate areas may be one of bounding boxes, pixelmasks, and polygon areas. The merged candidate area may then be of asame type. For example, if the first and second candidate areas arebounding boxes, the merged candidate area may also be a bounding box.

According to a second aspect, a non-transitory computer-readable storagemedium is provided having stored thereon instructions for implementingthe method according to the first aspect, when executed by a devicehaving processing capabilities.

The above-mentioned optional additional features of the method accordingto the first aspect, when applicable, apply to this second aspect aswell. In order to avoid undue repetition, reference is made to theabove.

According to a third aspect, a device for merging a first candidate arearelating to a candidate feature in a first image and a second candidatearea relating to a candidate feature in a second image is provided. Thefirst image and the second image have an overlapping region, and atleast a portion of the first candidate area and at least a portion ofthe second candidate area are located in the overlapping region. Thedevice comprises circuitry configured to execute an image overlap sizedetermining function configured to determine an image overlap sizeindicating a size of the overlapping region of the first image and thesecond image; a candidate area overlap ratio determining functionconfigured to determine a candidate area overlap ratio indicating anoverlap between the first candidate area and the second candidate area;a merging threshold determining function configured to determine amerging threshold based on the image overlap size; and a mergingfunction configured to, on condition that the candidate area overlapratio is larger than the merging threshold, merge the first candidatearea and the second candidate area, thereby forming a merged candidatearea.

The above-mentioned optional additional features of the method accordingto the first aspect, when applicable, apply to this third aspect aswell. In order to avoid undue repetition, reference is made to theabove.

A further scope of applicability will become apparent from the detaileddescription given below. However, it should be understood that thedetailed description and specific examples, while indicating preferred,are given by way of illustration only, since various changes andmodifications within the scope of the claims will become apparent tothose skilled in the art from this detailed description.

Hence, it is to be understood that is the concepts are not limited tothe particular component parts of the device described or acts of themethods described as such device and method may vary. It is also to beunderstood that the terminology used herein is for purpose of describingparticular embodiments only and is not intended to be limiting. It mustbe noted that, as used in the specification and the appended claim, thearticles “a,” “an,” “the,” and “said” are intended to mean that thereare one or more of the elements unless the context clearly dictatesotherwise. Thus, for example, reference to “a unit” or “the unit” mayinclude several devices, and the like. Furthermore, the words“comprising”, “including”, “containing” and similar wordings does notexclude other elements or steps.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects will now be described in more detail, withreference to appended figures. The figures should not be consideredlimiting but are instead used for explaining and understanding.

FIG. 1A shows a panoramic image captured using a fisheye lens and asingle image sensor.

FIG. 1B shows a so called quadview of the image of FIG. 1A includingfour transformed views of a respective one of four portions of the imageof FIG. 1A.

FIG. 1C shows a schematic illustration of two images having an overlap.

FIG. 2A shows an illustration of a first image and a second image and afirst candidate area and a second candidate area in relation to acandidate feature in the first image and the second image, respectively.

FIG. 2B shows a further illustration of a first image and a second imageand a first candidate area and a second candidate area in relation to acandidate feature in the first image and the second image, respectively.

FIG. 3 shows a flow chart in relation to embodiments of a method formerging a first candidate area relating to a candidate feature in afirst image and a second candidate area relating to a candidate featurein a second image.

FIG. 4 shows a schematic diagram in relation to embodiments of a devicemerging a first candidate area relating to a candidate feature in afirst image and a second candidate area relating to a candidate featurein a second image.

DETAILED DESCRIPTION

The concepts will now be described hereinafter with reference to theaccompanying drawings, in which currently preferred embodiments areillustrated. These concepts may, however, be embodied in many differentforms and should not be construed as limited to the embodiments setforth herein.

In the following, when referring to a candidate feature, such acandidate feature may relate to an object, a part of an object, asegment of an object, a portion where there is a movement or any otherfeature of an image in relation to which a candidate area has beenidentified in a preceding processing, such as object detection, instancesegmentation, panoptic segmentation, or motion based task to find wherein an image there is movement.

Furthermore, in the following, when referring to images, such images mayhave been captured by a respective one of image sensors or may relate toportions of a single image captured using a single image sensor. In thelatter case, the images may be transformations of the portions of thesingle image.

Furthermore, in the following, when referring to an overlapping regionor an overlap in relation to two images, such an overlapping region oran overlap is a region of one image of the two images that is includedalso in an the other image of the two images with or withouttransformation, or a region in one image of the two images that is arepresentation of a same portion of a scene as a region in the otherimage of the two images. In the former case, the two images may forexample be transformations of portions of a wide-angle view of a scenein order to reduce distortions of feature size relations. In the lattercase, the first and second images may be images captured by first andsecond image sensors, respectively, wherein the first and second imagesensors capture respective but partially overlapping portions of thescene.

The concepts herein are applicable when processing has been performedseparately in relation to a first image and a second image in order toidentify candidate areas relating to a candidate feature in the firstimage and a candidate feature in the second image, respectively. Forexample, the processing may relate to object detection wherein thecandidate feature to which each candidate area relates is an object.Such object detection may use a one-stage neural network such as a YouOnly Look Once (YOLO) architecture, Single Shot MultiBox Detector(SSD)/SSD-Lite, anchor free architectures such as CenterNet or FullyConvolutional One-Stage object detection (FCOS), and transformer-basednetworks such as DEtection TRansformer (DETR). The object detection mayfurther relate to two/multiple-stage detectors such as Region BasedConvolutional Neural Network (R-CNN), Faster R-CNN, etc. The objectdetection may further be Histogram of Oriented Gradients (HOG) based.For object detection output candidate areas may be boxes or other typeof polygons. The processing may further relate to segmentation, such asinstance segmentation (e.g., Mask R-CNN), and panoptic segmentation,wherein the candidate feature to which each candidate area relates is asegment. For segmentation output candidate areas may be polygons orpixel masks. The processing may further relate to motion-based taskswhere the task is to find where in an image there is movement, whereinthe candidate feature to which each candidate area relates is a portionin where there is movement. Output candidate areas from motion-basedtasks may be polygons or pixel masks.

When the above described or other processing has been performed,candidate features located partly in the overlapping region willtypically result in candidate areas in both of the first image and thesecond image. However, since the overlap between candidate areas indifferent images is limited by the size of the overlap, using a samemerging threshold as when merging candidate areas in the same image,will in some cases result in not merging the candidate boxes even thoughthe candidate areas do relate to the same candidate feature.

The concepts relate to determining when to merge candidate areas. Tothis end, Non-Maximum Suppression (NMS) is often applied. For example,most object detection approaches apply a sliding window over a featuremap and assigns foreground/background confidence values (scores)depending of the candidate features computed in that window. In somedeep learning detectors, such as single shot multibox detector, fasterRCNN detector, uses pre-defined anchor boxes in different resolution offeature layers. The neighbourhood windows (anchor boxes) have similarconfidence values to some extent and are considered as candidate areas.This leads to many candidate areas for a same candidate object(candidate feature). The NMS is used to filter the proposals based onIntersection over Union (IoU), which can be defined as the intersectionof two candidate areas divided by the union of the two candidate areas.

Each candidate area is compared with all the other candidate areas bycalculating the IoU of this candidate area with every other candidatearea. If the IoU is greater than a merging threshold N, then thecandidate areas are merged. One typical merge method is to select theproposal with highest confidence value.

In the following a first scenario will be described in relation to FIG.1A showing a panoramic image 110 captured using a fisheye lens, FIG. 1Bshowing a so called quadview of the image of FIG. 1A including fourtransformed views (images) 112, 114, 116, 118 of a respective one offour portions of the image 110 a of FIG. 1A, and FIG. 1C showing aschematic illustration of two images 120, 130 having an overlappingregion 140.

As can be seen from FIG. 1A, the use of a fisheye lens will provide awide angle view 110 but the view will be distorted such that a relativesize of a feature in the image will depend on the location in the view.When processing an image 110 as disclosed in FIG. 1A in order toidentify candidate areas in relation to candidate features in the image,some features may become relatively small due to the distortion andprocessing of the image may result in errors in relation to suchrelatively small features. Hence, instead of processing the image 110 asshown in FIG. 1A, the image may be divided into portions, such as fourportions. Furthermore, in order to reduce the distortion, each portionmay be transformed into what could be designated a part of adiscontinuous image or a separate image, such as the first to fourthimages (parts) 112, 114, 116, 118 in the quadview of FIG. 1B, such thatthe relative size of a feature is less dependent on its location in oneof the images 112, 114, 116, 118. Processing the images 112, 114, 116,118 separately will provide candidate areas in relation to candidatefeatures in the respective image of the images 112, 114, 116, 118.Furthermore, the image 110 in FIG. 1A is divided into portions and theportions are transformed into separate images such that the distortionis reduced and such that each image of the images 112, 114, 116, 118will include regions that is included also in other images of the images112, 114, 116, 118. As indicated hereinabove, a region that is includedin two images of the images 112, 114, 116, 118 is referred to as anoverlapping region of the two images. A schematic illustration of afirst image 120 and a second image 130 having an overlapping region 140is shown in FIG. 1C. The first image 120 and the second image 130 ofFIG. 1C may correspond to pairs of images in FIG. 1B, such as the firstimage 112 and the second image 114, the first image 112 and the thirdimage 116, the second image 114 and the fourth image 118, and the thirdimage 116 and the fourth image 118. It is to be noted that FIG. 1C onlyprovides a schematic illustration of how a feature 150 in the form of anobject may have a first portion that is located only in the first image120, a second portion that is located only in the second image 130 and athird portion that is located in the overlapping region 140 which meansthat it is located both in the first image 120 and in the second image130. For example, the overlapping region 140 between the first image 120and the second image 130 in FIG. 1C is rectangular. In the actual case,an overlapping region of two images of the four images 112, 114, 116,118 of the quadview of FIG. 1B would typically not be rectangular.However, the respective overlapping region of images of the four images112, 114, 116, 118 of the quadview of FIG. 1B can be determined based onknowledge of how the four images 112, 114, 116, 118 of the quadview ofFIG. 1B were created from the image 110 of FIG. 1A, such as therespective portion of the image 110 of FIG. 1A used for each image ofthe four images 112, 114, 116, 118 of the quadview of FIG. 1B and thetransformation used etc.

In order to ensure proper processing to identify candidate areas,division of an original image into portions and transformation of theportions into separate images, should preferably be configured such thatthere is no region of the original image that is not shown in any one ofthe separate images. This is not the case in relation to the originalimage 110 in FIG. 1A and the separate images 112, 114, 116, 118 in FIG.1B.

It is to be noted, that the quadview of FIG. 1B may also be created andused for other reasons, such as in order to provide a better overview ofthe scene when one or more images or a video of the scene is viewed.

In the following a second scenario will be described in relation to FIG.1C showing a schematic illustration of two images having an overlap. Asalternative of using a wide angle lens, such as a fisheye lens, apanoramic image covering a wide angle can be produced by using two imagesensors. In the second scenario, the first image 120 would be an imagecaptured by a first image sensor and the second image 130 would be animage captured by a second image sensor. The overlapping region 140would be a region captured by both by the first image sensor and by thesecond image sensor. In contrast to the first scenario, the overlappingregion would hence not relate to a region in both the first image andthe second image captured by the same image sensor and, hence, notinclude identical image data. Instead, it would include image data fromthe respective image sensor of the first image sensor and the secondimage sensor representing a same portion of the scene captured by thefirst image sensor and the second image sensor. In this second scenario,the first image 120 and the second image 130 may later be stitchedtogether to form a single combined image showing a representation of thescene captured by the first image sensor and the second image sensor.However, in this second scenario also, when processing to identifycandidate areas, e.g., using one of the types of processing as describedhereinabove, it may be advantageous to process the first image 120 andthe second image 130 separately. For example, the first image sensor andthe second image sensor may be co-located or otherwise related to arespective one of two processing units and thus, the processing of thefirst image 120 and the second image 130 may be performed in parallel toprovide resulting candidate areas from the processing within shortertime. Also, by processing the first image 120 and the second image 130separately, will result in processing two smaller images which willenable identification of smaller candidate areas, e.g., by enablingdetection of smaller objects. There may also be hardware restrictionspreventing processing of a combination of the first image 120 and thesecond image 130 as a combined image.

In the different scenarios and in different specific implementations, anoverlapping region between two images may be of different sizes asreflected in FIG. 2A and FIG. 2B. FIG. 2A shows an illustration of afirst image 220 a and a second image 230 a having an overlapping area240 a. Furthermore, in FIG. 2A, a first candidate area 260 a in the formof a bounding box is shown in a dashed line and a second candidate area270 a in the form of a bounding box is shown in a dotted line inrelation to a candidate feature 250 a in the form of an object in thefirst image 220 a and the second image 230 a, respectively. FIG. 2Bshows a further illustration of a first image 220 b and a second image230 b having an overlapping area 240 b. Furthermore, in FIG. 2B a firstcandidate area 260 b in the form of a bounding box is shown in a dashedline and a second candidate area 270 b in the form of a bounding box isshown in a dotted line in relation to a candidate feature 250 b in theform of an object in the first image 220 b and the second image 230 b,respectively. As can be seen from the illustrations of FIG. 2A and FIG.2B, the overlapping region 240 a of the first image 220 a and the secondimage 230 a in FIG. 2A is smaller than an overlapping region 240 b ofthe first image 220 b and the second image 230 b in FIG. 2B.Furthermore, a size of an intersection between a candidate area in oneof the images and a candidate area in the other if the images is limitedby the size of the overlapping region, the size of the intersectionbetween candidate areas for a feature of which at least a portion islocated in an overlapping region will depend on the size of theoverlapping region. As can be seen from the illustrations of FIG. 2A andFIG. 2B, the intersection between the candidate area 260 a in the firstimage 220 a and the candidate area 270 a in the second image 230 a ofFIG. 2A is smaller than the intersection between the candidate area 260b in the first image 220 b and the candidate area 270 b in the secondimage 230 b of FIG. 2B since the overlapping region 240 a in FIG. 2A issmaller than the overlapping region in FIG. 2B. On the other hand, asize of a union between a candidate area in one of the images and acandidate area in the other of the images is generally not significantlyaffected by the size of the overlapping region but rather depends on thesize of a feature to which they relate in the case the candidate areasin the images in fact relate to a same feature of which at least aportion is located in the overlapping region.

FIG. 2A and FIG. 2B each discloses two candidate areas in the form ofrectangular bounding boxes, which candidate areas each have a portion inthe overlapping region. The merging of candidate areas of the presentdisclosure is applicable mutatis mutandis for candidate areas of othertypes and shapes, such as pixel masks and polygon shaped as describedhereinabove. Furthermore, the two candidate areas in FIG. 2A and FIG. 2Brelate to a feature in the form of an object. However, the merging ofcandidate areas of the present disclosure is applicable mutatis mutandisfor candidate areas relating to features of other types and being theresult of other types of processing, such as segments resulting fromsegmentation of different types as described hereinabove and portionswhere there is motion resulting from motion-based tasks as describedhereinabove.

Embodiments of a method 300 for merging a first candidate area 260 a,260 b relating to a candidate feature 250, 250 a, 250 b in a first image220, 220 a, 220 b and a second candidate area 270 a, 270 b relating to acandidate feature 250, 250 a, 250 b in a second image 230, 230 a, 230 bwill now be described in relation to FIG. 3 showing a flow chart 300 andwith reference to FIG. 1A, FIG. 1B, FIG. 1C, FIG. 2A and FIG. 2B. Thefirst image 220, 220 a, 220 b and the second image 230, 230 a, 230 bhave an overlapping region 240, 240 a, 240 b, and at least a portion ofthe first candidate area 260 a, 260 b and at least a portion of thesecond candidate area 270 a, 270 b are located in the overlapping region240 a, 240 b.

The first image 220, 220 a, 220 b and the second image 230, 230 a, 230 bmay relate to any scenario resulting in them to have an overlappingregion 240, 240 a, 240 b.

For example, the first image 220, 220 a, 220 b may be a firsttransformed view of a first portion of a fisheye image 110, i.e., apanoramic image 110 captured by means of a fisheye lens, and the secondimage 230, 230 a, 230 b may be a second transformed view of a secondportion of the fisheye image 110 as described in relation to FIG. 1A andFIG. 1B. The first candidate area 260 a, 260 b and the second candidatearea 270 a, 270 b may, in this example, be identified in coordinates ofthe image 110. For example, if candidate areas have been identifiedbased on previous processing to identify candidate features in the firsttransformed view (image) 112 and in the second transformed view (image)114 of the quadview in FIG. 1B, the candidate areas may first betransformed to coordinates of the image 110 of FIG. 1A by means of aninverse of the transform used to transform the corresponding portions ofthe image 110 to the first transformed view (image) 112 and in thesecond transformed view (image) 114. Hence, even if the first candidatearea 260 a, 260 b relates to a candidate feature 250, 250 a, 250 b inthe first image 220, 220 a, 220 b and the second candidate area 270 a,270 b relates to a candidate feature 250, 250 a, 250 b in the secondimage 230, 230 a, 230 b, this does not necessarily mean that the firstcandidate area 260 a, 260 b and the second candidate area 270 a, 270 bare identified in the coordinates of the first image 220, 220 a, 220 band the second image 230, 230 a, 230 b, respectively. It is to be notedthat the first candidate area 260 a, 260 b and the second candidate area270 a, 270 b in FIG. 2A and FIG. 2B are rectangular bounding boxes.However, in the scenario in relation to FIG. 1A and FIG. 1B, suchbounding boxes would relate to transformations of rectangular boundingboxes identified in the four transformed views (images) 112, 114, 116,118 into the coordinates of the image 110.

As another example, the first image 220, 220 a, 220 b may be an imagecaptured by a first image sensor and the second image 230, 230 a, 230 bmay be an image captured by a second image sensor as disclosed inrelation to FIG. 1C. The first candidate area 260 a, 260 b and thesecond candidate area 270 a, 270 b may, in this example, be identifiedin coordinates of a combined image comprising the first image 220, 220a, 220 b and the second image 230, 230 a, 230 b.

The first candidate area 260 a, 260 b and the second candidate area 270a, 270 b may be any candidate areas resulting from previous processingto identify candidate features in the first image 220, 220 a, 220 b andthe second image 270 a, 270 b, respectively, such as processing of thetypes described hereinabove. Depending on scenario, the first candidatearea 260 a, 260 b and the second candidate area 270 a, 270 b may betransformations of candidate areas identified in the first image 220,220 a, 220 b and in the second image 230, 230 a, 230 b, respectively.

Furthermore, the first candidate area 260 a, 260 b and the secondcandidate area 270 a, 270 b may be candidate areas resulting directlyfrom previous processing to identify candidate features in the firstimage 220, 220 a, 220 b and the second image 270 a, 270 b, with orwithout transformation depending on scenario. In alternative, the firstcandidate area 260 a, 260 b and the second candidate area 270 a, 270 bmay be candidate areas resulting from first merging candidate areas inthe respective image separately based on a candidate area mergingalgorithm, such as NMS as described hereinabove. Hence, for example, thefirst candidate area 260 a, 260 b may relate to two or more candidateareas identified in the first image 220, 220 a, 220 b being merged bymeans of a merging algorithm applied only to candidate areas in thefirst image 220, 220 a, 220 b, i.e., without consideration of anycandidate areas identified in the second image 230, 230 a, 230 b.

The first candidate area 260 a, 260 b and the second candidate area 270a, 270 b may be bounding boxes, pixel masks, or polygon areas asdescribed hereinabove. The shape of the first candidate area 260 a, 260b and the second candidate area 270 a, 270 b depends on the type ofprocessing used to identify them, such as the types of processingdescribed hereinabove. Furthermore, the shape of the first candidatearea 260 a, 260 b and the second candidate area 270 a, 270 b may alsodepend on a transformation performed of corresponding candidate areasidentified in the first image 220, 220 a, 220 b and the second image230, 230 a, 230 b, respectively, when in the first image 220, 220 a, 220b and the second image 230, 230 a, 230 b are transformed portions of asingle image 110.

In the method 300 an image overlap size indicating a size of theoverlapping region 240, 240 a, 240 b of the first image 220, 220 a, 220b and the second image 230, 230 a, 230 b is determined S310. The way thesize is determined will typically depend on the shape of the overlappingregion. For example, for an overlapping region 240, 240 a, 240 b, asdisclosed in FIG. 1C, FIG. 2A and FIG. 2B, which is rectangular and hasa uniform width, the image overlap size may be set to the width of theoverlap.

Furthermore, a candidate area overlap ratio indicating a ratio ofoverlap between the first candidate area 260 a, 260 b and the secondcandidate area 270 a, 270 b is determined S320. Any kind ofnormalization in relation to sizes of the first candidate area 260 a,260 b and the second candidate area 270 a, 270 b may be used andtypically in relation to the size of the union of the first candidatearea 260 a, 260 b and the second candidate area 270 a, 270 b. Forexample, the candidate area overlap ratio may be determined as theintersection of the first candidate area 260 a, 260 b and the secondcandidate area 270 a, 270 b divided by the union of the first candidatearea 260 a, 260 b and the second candidate area 270 a, 270 b.

Then a merging threshold is determined S340 based on the image overlapsize. The merging threshold may be determined to be increasing with theimage overlap size. For example, the merging threshold may be determinedto be proportional to the image overlap size. Hence, for a given overlapsize, the merging threshold will be determined to be the given overlapsize multiplied with a proportionality constant.

On condition C335 that the candidate area overlap ratio is larger thanthe merging threshold, the first candidate area 260 a, 260 b and thesecond candidate area 270 a, 270 b are merged S340 to form a mergedcandidate area. The merged candidate area may consist of the union ofthe first candidate area 260 a, 260 b and the second candidate area 270a, 270 b.

By determining the merging threshold based on the image overlap size therisk of not merging the first candidate area and the second candidatearea even when they relate to the same feature is reduced. Hence, therisk is reduced that the same feature in separate images risk beidentified as two separate features.

The method 300 may be adapted to better accommodate different shapes andorientations of features resulting in identification of different shapesand orientation of candidate areas. For example, if a feature iselongated, different orientations of the feature will result indifferent proportions of the feature and hence different proportions ofcandidate areas identified will typically be in an overlapping region.For example, in relation to FIG. 2A and FIG. 2B, if the feature 250, 250a, 250 b would have been horizontal instead of vertical, a smallerproportion of the feature would have been in the overlapping region 240,240 a, 240 b. Hence, to take this into account, a size of theintersection of the overlapping region 240, 240 a, 240 b and the unionof the first candidate area 260 a, 260 b and the second candidate area270 a, 270 b may be determined. The merging threshold may be determinedbased on the image overlap size by basing the merging threshold on thesize of the intersection of the overlapping region 240, 240 a, 240 b andthe union of the first candidate area 260 a, 260 b and the secondcandidate area 270 a, 270 b. For example, the merging threshold may bedetermined to be proportional to the size of the intersection of theoverlapping region 240, 240 a, 240 b and the union of the firstcandidate area 260 a, 260 b and the second candidate area 270 a, 270 b.Hence, for a given the size of the intersection of the overlappingregion 240, 240 a, 240 b and the union of the first candidate area 260a, 260 b and the second candidate area 270 a, 270 b, the mergingthreshold will be determined to be the given the size of theintersection of the overlapping region 240, 240 a, 240 b and the unionof the first candidate area 260 a, 260 b and the second candidate area270 a, 270 b multiplied with a proportionality constant. This adaptationis further suitable also when an overlapping region does not have auniform width.

Even though embodiments of the method 300 have been described inrelation to two images, the embodiments of the method are alsoapplicable to cases where there are three or more images. In such acase, embodiments of the method 300 may be applied for the three or moreimages in one step for all images or recursively, i.e., first for firsttwo images having an overlap, then for the result from the first twoimages and a further image of the three or more images having an overlapwith either of the two first images, and so on.

FIG. 4 shows a schematic diagram in relation to embodiments of a device400 of the present disclosure for merging a first candidate arearelating to a candidate feature in a first image and a second candidatearea relating to a candidate feature in a second image, wherein thefirst image and the second image have an overlapping region, and whereinat least a portion of the first candidate area and at least a portion ofthe second candidate area are located in the overlapping region. Thedevice 400 may for example be a camera. The device 400 comprisescircuitry 410. The circuitry 410 is configured to carry out thefunctions 432, 434, 436, 438 of the device 400. The circuitry 410 mayinclude a processor 412, such as a central processing unit (CPU),microcontroller, or microprocessor. The processor 412 is configured toexecute program code. The program code may for example be configured tocarry out the functions 432, 434, 436, 438 of the device 400.

The device 400 may further comprise a memory 430. The memory 430 may beone or more of a buffer, a flash memory, a hard drive, a removablemedia, a volatile memory, a non-volatile memory, a random access memory(RAM), or another suitable device. In a typical arrangement, the memory430 may include a non-volatile memory for long term data storage and avolatile memory that functions as system memory for the circuitry 410.The memory 430 may exchange data with the circuitry 410 over a data bus.Accompanying control lines and an address bus between the memory 430 andthe circuitry 410 also may be present.

The functions 432, 434, 436, 438 of the device 400 may be embodied inthe form of executable logic routines (e.g., lines of code, softwareprograms, etc.) that are stored on a non-transitory computer readablemedium (memory) 430 of the device 400 and are executed by the circuitry410, e.g., using the processor 412 in the circuitry 410. Furthermore,the functions 432, 434, 436, 438 of the device 400 may be a stand-alonesoftware application or form a part of a software application. Thedescribed functions may be considered a method that a processing unit,e.g., the processor 412 of the circuitry 410 is configured to carry out.Also, while the described functions 432, 434, 436, 438 may beimplemented in software, such functionality may as well be carried outvia dedicated hardware or firmware, or some combination of hardware,firmware and/or software.

The circuitry 410 is configured to execute an image overlap sizedetermining function 432 configured to determine an image overlap sizeindicating a size of the overlapping region of the first image and thesecond image.

The circuitry 410 is further configured to execute a candidate areaoverlap ratio determining function 434 configured to determine acandidate area overlap ratio indicating an overlap between the firstcandidate area and the second candidate area

The circuitry 410 is further configured to execute a merging thresholddetermining function 436 configured to determine a merging thresholdbased on the image overlap size.

The circuitry 410 is further configured to execute a merging function438 configured to, on condition that the candidate area overlap ratio islarger than the merging threshold, merge the first candidate area andthe second candidate area, thereby forming a merged candidate area.

The device 400 and the functions 432, 434, 436, 438 carried out by thecircuitry 410 may be further adapted as the method 300 and thecorresponding steps of the method 300 described in relation to FIG. 3,respectively.

A person skilled in the art realizes that the present concepts are notlimited to the embodiments described above. On the contrary, manymodifications and variations are possible within the scope of theappended claims. Such modifications and variations can be understood andeffected by a skilled person in practicing the concepts, from a study ofthe drawings, the disclosure, and the appended claims.

1. A method for merging a first candidate area relating to a candidatefeature in a first image and a second candidate area relating to acandidate feature in a second image, wherein the first image and thesecond image have an overlapping region, and wherein at least a portionof the first candidate area and at least a portion of the secondcandidate area are located in the overlapping region, the methodcomprising: determining an image overlap size indicating a size of theoverlapping region of the first image and the second image; determininga candidate area overlap ratio indicating a ratio of overlap between thefirst candidate area and the second candidate area; determining amerging threshold based on the image overlap size; and on condition thatthe candidate area overlap ratio is larger than the merging threshold,merging the first candidate area and the second candidate area, therebyforming a merged candidate area.
 2. The method according to claim 1,wherein the merging threshold is determined to be proportional to theimage overlap size.
 3. The method according to claim 1, wherein thecandidate area overlap ratio is determined as the intersection of thefirst candidate area and the second candidate area divided by the unionof the first candidate area and the second candidate area.
 4. The methodaccording to claim 1, wherein the first candidate area and the secondcandidate area are identified in coordinates of a combined imagecomprising the first image and the second image.
 5. The method accordingto claim 1, wherein the first image is a first transformed view of afirst portion of a fisheye image and the second image is a secondtransformed view of a second portion of the fisheye image, and whereinthe first candidate area and the second candidate area are identified incoordinates of the fisheye image.
 6. The method according to claim 1,wherein the merged candidate area consists of the union of the firstcandidate area and the second candidate area.
 7. The method according toclaim 1, wherein the first and second candidate areas are one ofbounding boxes, pixel masks, and polygon areas.
 8. A non-transitorycomputer readable storage medium having stored thereon instructions forimplementing a method, when executed on a device having processingcapabilities, the method for merging a first candidate area relating toa candidate feature in a first image and a second candidate arearelating to a candidate feature in a second image, wherein the firstimage and the second image have an overlapping region, and wherein atleast a portion of the first candidate area and at least a portion ofthe second candidate area are located in the overlapping region, themethod comprising: determining an image overlap size indicating a sizeof the overlapping region of the first image and the second image;determining a candidate area overlap ratio indicating a ratio of overlapbetween the first candidate area and the second candidate area;determining a merging threshold based on the image overlap size; and oncondition that the candidate area overlap ratio is larger than themerging threshold, merging the first candidate area and the secondcandidate area, thereby forming a merged candidate area.
 9. Thenon-transitory computer readable storage medium method according toclaim 8, wherein the merging threshold is determined to be proportionalto the image overlap size.
 10. The non-transitory computer readablestorage medium method according to claim 8, wherein the candidate areaoverlap ratio is determined as the intersection of the first candidatearea and the second candidate area divided by the union of the firstcandidate area and the second candidate area.
 11. The non-transitorycomputer readable storage medium method according to claim 8, whereinthe first candidate area and the second candidate area are identified incoordinates of a combined image comprising the first image and thesecond image.
 12. The non-transitory computer readable storage mediummethod according to claim 8, wherein the first image is a firsttransformed view of a first portion of a fisheye image and the secondimage is a second transformed view of a second portion of the fisheyeimage, and wherein the first candidate area and the second candidatearea are identified in coordinates of the fisheye image.
 13. Thenon-transitory computer readable storage medium method according toclaim 8, wherein the merged candidate area consists of the union of thefirst candidate area and the second candidate area.
 14. A device formerging a first candidate area relating to a candidate feature in afirst image and a second candidate area relating to a candidate featurein a second image, wherein the first image and the second image have anoverlapping region, and wherein at least a portion of the firstcandidate area and at least a portion of the second candidate area arelocated in the overlapping region, the device comprising: circuitryconfigured to execute: an image overlap size determining functionconfigured to determine an image overlap size indicating a size of theoverlapping region of the first image and the second image; a candidatearea overlap ratio determining function configured to determine acandidate area overlap ratio indicating an overlap between the firstcandidate area and the second candidate area; a merging thresholddetermining function configured to determine a merging threshold basedon the image overlap size; and a merging function configured to, oncondition that the candidate area overlap ratio is larger than themerging threshold, merge the first candidate area and the secondcandidate area, thereby forming a merged candidate area.
 15. The deviceaccording to claim 14, wherein the merging threshold determiningfunction is configured to determine merging threshold to be proportionalto the image overlap size.
 16. The device according to claim 14, whereinthe candidate area overlap ratio determining function is configured todetermine the candidate area overlap ratio as the intersection of thefirst candidate area and the second candidate area divided by the unionof the first candidate area and the second candidate area.
 17. Thedevice according to claim 14, wherein the first candidate area andsecond candidate area are identified in coordinates of a combined imagecomprising the first image and the second image.
 18. The deviceaccording to claim 14, wherein the first image is a first transformedview of a first portion of a fisheye image and the second image is asecond transformed view of a second portion of the fisheye image, andwherein the first candidate area and second candidate area areidentified in coordinates of the fisheye image.
 19. The device accordingto claim 14, wherein the merged candidate area consists of the union ofthe first candidate area and the second candidate area.
 20. The deviceaccording to claim 9, wherein the first and second candidate areas areone of bounding boxes, pixel masks, and polygon areas.